ZEPPELIN-598 ] Dynamic loading for Interpreter and API #908
Conversation
|
I tested this funtionality but I got failed. [My test process]
but I got an exception and md is not loaded. Did i something wrong? |
|
Hi @cloverhearts, I hope this docs can help any others who want to review this PR :) |
|
Thank you @AhyoungRyu fixed. |
|
done. |
|
Thank you, for tested. @astroshim |
|
@cloverhearts I tested and really nice working. Thanks. |
|
cool - should we have some sort of access control for this later? loading a jar by POST request on a server seems rather open ;) |
|
@felixcheung Thank you for a good opinion. |
|
Ping. I make a new issue that may break this PR - https://issues.apache.org/jira/browse/ZEPPELIN-932 -. Could anyone please check the status of this PR? |
conf/zeppelin-env.sh.template
Outdated
| # export ZEPPELIN_IDENT_STRING # A string representing this instance of zeppelin. $USER by default. | ||
| # export ZEPPELIN_NICENESS # The scheduling priority for daemons. Defaults to 0. | ||
| # export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for interpreter's additional dependency loading | ||
| # export ZEPPELIN_INTERPRETER_DOWNLOAD_DIR # download interpreter for directory path |
There was a problem hiding this comment.
I think something like "Path for interpreter download" is more clear
|
@jongyoul |
| try { | ||
| urls = recursiveBuildLibList(interpreterDir); | ||
| } catch (MalformedURLException e1) { | ||
| logger.error("Can't load jars ", e1); |
There was a problem hiding this comment.
should it abort when it fails? or is this more a warning?
There was a problem hiding this comment.
@felixcheung
Thank you for your opinion.
i've changed.
Conflicts: conf/zeppelin-env.cmd.template conf/zeppelin-env.sh.template
|
rebase and fix |
|
Merge master branch into this pr. |
|
@cloverhearts exciting changes! This would help us to make convenience binary for a release smaller as well |
|
No luck, it can't get to load my jar from my local repo (nexus on docker). My request : { The java trace (extract at the end) shows that my local repo seems to be ignored and that of course, it can't find my jar in official repo ;-) Any advice? INFO [2016-06-21 22:57:37,461]({qtp922151033-19} InterpreterRestApi.java[loadDynamicInterpreter]:262) - dynamic load interpreter interpreterGroupName [jdbc] name [neo4j] |
### What is this PR for? This PR is to automate release publish to maven repository. We used to use maven-deploy-plugin and maven-release-plugin for release but somehow it didn't work well with Zeppelin so 0.5.5 and 0.5.6 haven't been published to maven repository. Publishing release to maven repository will eventually help zeppelin to reduce binary package size by leading users to use Dynamic interpreter loading(#908). Originally below modules were skipped for maven release - all interpreters(except spark) - zeppelin-display - zeppelin-server - zeppelin-distribution on the other hand this pr will skip only: - zeppelin-distribution ### What type of PR is it? Infra ### Todos - [x] Include SparkR/R interpreter in release - [x] Create common_release.sh to remove build configuration duplication - [x] Check curl networking failure ### What is the Jira issue? [ZEPPELIN-986](https://issues.apache.org/jira/browse/ZEPPELIN-986) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes, https://cwiki.apache.org/confluence/display/ZEPPELIN/Preparing+Zeppelin+Release will be updated accordingly once this pr is merged. Author: Mina Lee <[email protected]> Closes #994 from minahlee/ZEPPELIN-986 and squashes the following commits: b0e8e67 [Mina Lee] Revert "Add geode, scalding profile in maven artifact build" cd4cbcd [Mina Lee] curl failure check c0ea07c [Mina Lee] Fix wrong indentation a88bc1d [Mina Lee] Add geode, scalding profile in maven artifact build 2cced61 [Mina Lee] Add r to binary package and maven build 903bc12 [Mina Lee] Move duplicate code to common_release.sh a3eb676 [Mina Lee] Include zeppelin-server module in publish artifiact 48d338f [Mina Lee] Rollback mistakenly removed plugin aafaf42 [Mina Lee] Follow google shell style guide 30dcc65 [Mina Lee] remove deploy plugin from pom since custom script will be used instead for deploy cd1f08c [Mina Lee] Refactor create release script e764f5f [Mina Lee] Add maven publish release script
### What is this PR for? This PR is to automate release publish to maven repository. We used to use maven-deploy-plugin and maven-release-plugin for release but somehow it didn't work well with Zeppelin so 0.5.5 and 0.5.6 haven't been published to maven repository. Publishing release to maven repository will eventually help zeppelin to reduce binary package size by leading users to use Dynamic interpreter loading(#908). Originally below modules were skipped for maven release - all interpreters(except spark) - zeppelin-display - zeppelin-server - zeppelin-distribution on the other hand this pr will skip only: - zeppelin-distribution ### What type of PR is it? Infra ### Todos - [x] Include SparkR/R interpreter in release - [x] Create common_release.sh to remove build configuration duplication - [x] Check curl networking failure ### What is the Jira issue? [ZEPPELIN-986](https://issues.apache.org/jira/browse/ZEPPELIN-986) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes, https://cwiki.apache.org/confluence/display/ZEPPELIN/Preparing+Zeppelin+Release will be updated accordingly once this pr is merged. Author: Mina Lee <[email protected]> Closes #994 from minahlee/ZEPPELIN-986 and squashes the following commits: b0e8e67 [Mina Lee] Revert "Add geode, scalding profile in maven artifact build" cd4cbcd [Mina Lee] curl failure check c0ea07c [Mina Lee] Fix wrong indentation a88bc1d [Mina Lee] Add geode, scalding profile in maven artifact build 2cced61 [Mina Lee] Add r to binary package and maven build 903bc12 [Mina Lee] Move duplicate code to common_release.sh a3eb676 [Mina Lee] Include zeppelin-server module in publish artifiact 48d338f [Mina Lee] Rollback mistakenly removed plugin aafaf42 [Mina Lee] Follow google shell style guide 30dcc65 [Mina Lee] remove deploy plugin from pom since custom script will be used instead for deploy cd1f08c [Mina Lee] Refactor create release script e764f5f [Mina Lee] Add maven publish release script (cherry picked from commit 85d7057) Signed-off-by: Mina Lee <[email protected]>
|
@ggdupont |
|
What's the status of this PR? |
|
@jongyoul Task is waiting. |
|
@cloverhearts I've got it. Thanks. |
8 participants
What is this PR for?
Use of external libraries that are included in the Roadmap,
The interpreter needs related to dynamic loading capabilities.
What type of PR is it?
Feature
Todos
ScreenShots
Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-598
How should this be tested?
By annotating the interpreter setting items below, try using the loadDynamicInterpreter method.
Or create a new interpreter loads created in the local Maven repository.
Step 1.
markdown interpreter to local distribute
Step 2.
Comment out the markdown
incubator-zeppelin/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
Step 3.
and build.
For example.
Step 4.
remove to interpreter/md
option.
fixed conf/zeppelin-env.sh(cmd) download interpreter directory.
Step 5.
and API Call (load interpreter) example load markdown
Step 6.
In interperter menu for zeppelin web, you can new setting information for md.
Questions: